{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Basic Metrics\n",
    "\n",
    "When we think about summarizing data, what are the metrics that we look at?\n",
    "\n",
    "In this notebook, we will look in the price of weed dataset along with the demographic information of the United States. \n",
    "\n",
    "To read how the data was acquired, please read [this](https://github.com/amitkaps/weed/blob/master/1-Acquire.ipynb) to get more information\n",
    "\n",
    "This notebook will make use of pandas quite a bit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from datetime import datetime as dt\n",
    "from scipy import stats"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Read the input datasets. There are three datasets:\n",
    "\n",
    "1. Weed price by date / state\n",
    "2. Demographics of State\n",
    "3. Population of state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "prices_pd = pd.read_csv(\"../data/Weed_Price.csv\", parse_dates=[-1])\n",
    "demography_pd = pd.read_csv(\"../data/Demographics_State.csv\")\n",
    "population_pd = pd.read_csv(\"../data/Population_State.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State</th>\n",
       "      <th>HighQ</th>\n",
       "      <th>HighQN</th>\n",
       "      <th>MedQ</th>\n",
       "      <th>MedQN</th>\n",
       "      <th>LowQ</th>\n",
       "      <th>LowQN</th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Alabama</td>\n",
       "      <td>339.06</td>\n",
       "      <td>1042</td>\n",
       "      <td>198.64</td>\n",
       "      <td>933</td>\n",
       "      <td>149.49</td>\n",
       "      <td>123</td>\n",
       "      <td>2014-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Alaska</td>\n",
       "      <td>288.75</td>\n",
       "      <td>252</td>\n",
       "      <td>260.60</td>\n",
       "      <td>297</td>\n",
       "      <td>388.58</td>\n",
       "      <td>26</td>\n",
       "      <td>2014-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Arizona</td>\n",
       "      <td>303.31</td>\n",
       "      <td>1941</td>\n",
       "      <td>209.35</td>\n",
       "      <td>1625</td>\n",
       "      <td>189.45</td>\n",
       "      <td>222</td>\n",
       "      <td>2014-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Arkansas</td>\n",
       "      <td>361.85</td>\n",
       "      <td>576</td>\n",
       "      <td>185.62</td>\n",
       "      <td>544</td>\n",
       "      <td>125.87</td>\n",
       "      <td>112</td>\n",
       "      <td>2014-01-01</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>California</td>\n",
       "      <td>248.78</td>\n",
       "      <td>12096</td>\n",
       "      <td>193.56</td>\n",
       "      <td>12812</td>\n",
       "      <td>192.92</td>\n",
       "      <td>778</td>\n",
       "      <td>2014-01-01</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        State   HighQ  HighQN    MedQ  MedQN    LowQ  LowQN       date\n",
       "0     Alabama  339.06    1042  198.64    933  149.49    123 2014-01-01\n",
       "1      Alaska  288.75     252  260.60    297  388.58     26 2014-01-01\n",
       "2     Arizona  303.31    1941  209.35   1625  189.45    222 2014-01-01\n",
       "3    Arkansas  361.85     576  185.62    544  125.87    112 2014-01-01\n",
       "4  California  248.78   12096  193.56  12812  192.92    778 2014-01-01"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prices_pd.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State</th>\n",
       "      <th>HighQ</th>\n",
       "      <th>HighQN</th>\n",
       "      <th>MedQ</th>\n",
       "      <th>MedQN</th>\n",
       "      <th>LowQ</th>\n",
       "      <th>LowQN</th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>22894</th>\n",
       "      <td>Virginia</td>\n",
       "      <td>364.98</td>\n",
       "      <td>3513</td>\n",
       "      <td>293.12</td>\n",
       "      <td>3079</td>\n",
       "      <td>NaN</td>\n",
       "      <td>284</td>\n",
       "      <td>2014-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22895</th>\n",
       "      <td>Washington</td>\n",
       "      <td>233.05</td>\n",
       "      <td>3337</td>\n",
       "      <td>189.92</td>\n",
       "      <td>3562</td>\n",
       "      <td>NaN</td>\n",
       "      <td>160</td>\n",
       "      <td>2014-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22896</th>\n",
       "      <td>West Virginia</td>\n",
       "      <td>359.35</td>\n",
       "      <td>551</td>\n",
       "      <td>224.03</td>\n",
       "      <td>545</td>\n",
       "      <td>NaN</td>\n",
       "      <td>60</td>\n",
       "      <td>2014-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22897</th>\n",
       "      <td>Wisconsin</td>\n",
       "      <td>350.52</td>\n",
       "      <td>2244</td>\n",
       "      <td>272.71</td>\n",
       "      <td>2221</td>\n",
       "      <td>NaN</td>\n",
       "      <td>167</td>\n",
       "      <td>2014-12-31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22898</th>\n",
       "      <td>Wyoming</td>\n",
       "      <td>322.27</td>\n",
       "      <td>131</td>\n",
       "      <td>351.86</td>\n",
       "      <td>197</td>\n",
       "      <td>NaN</td>\n",
       "      <td>12</td>\n",
       "      <td>2014-12-31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "               State   HighQ  HighQN    MedQ  MedQN  LowQ  LowQN       date\n",
       "22894       Virginia  364.98    3513  293.12   3079   NaN    284 2014-12-31\n",
       "22895     Washington  233.05    3337  189.92   3562   NaN    160 2014-12-31\n",
       "22896  West Virginia  359.35     551  224.03    545   NaN     60 2014-12-31\n",
       "22897      Wisconsin  350.52    2244  272.71   2221   NaN    167 2014-12-31\n",
       "22898        Wyoming  322.27     131  351.86    197   NaN     12 2014-12-31"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prices_pd.tail()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>region</th>\n",
       "      <th>total_population</th>\n",
       "      <th>percent_white</th>\n",
       "      <th>percent_black</th>\n",
       "      <th>percent_asian</th>\n",
       "      <th>percent_hispanic</th>\n",
       "      <th>per_capita_income</th>\n",
       "      <th>median_rent</th>\n",
       "      <th>median_age</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>alabama</td>\n",
       "      <td>4799277</td>\n",
       "      <td>67</td>\n",
       "      <td>26</td>\n",
       "      <td>1</td>\n",
       "      <td>4</td>\n",
       "      <td>23680</td>\n",
       "      <td>501</td>\n",
       "      <td>38.1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>alaska</td>\n",
       "      <td>720316</td>\n",
       "      <td>63</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>6</td>\n",
       "      <td>32651</td>\n",
       "      <td>978</td>\n",
       "      <td>33.6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>arizona</td>\n",
       "      <td>6479703</td>\n",
       "      <td>57</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>30</td>\n",
       "      <td>25358</td>\n",
       "      <td>747</td>\n",
       "      <td>36.3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>arkansas</td>\n",
       "      <td>2933369</td>\n",
       "      <td>74</td>\n",
       "      <td>15</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>22170</td>\n",
       "      <td>480</td>\n",
       "      <td>37.5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>california</td>\n",
       "      <td>37659181</td>\n",
       "      <td>40</td>\n",
       "      <td>6</td>\n",
       "      <td>13</td>\n",
       "      <td>38</td>\n",
       "      <td>29527</td>\n",
       "      <td>1119</td>\n",
       "      <td>35.4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       region  total_population  percent_white  percent_black  percent_asian  \\\n",
       "0     alabama           4799277             67             26              1   \n",
       "1      alaska            720316             63              3              5   \n",
       "2     arizona           6479703             57              4              3   \n",
       "3    arkansas           2933369             74             15              1   \n",
       "4  california          37659181             40              6             13   \n",
       "\n",
       "   percent_hispanic  per_capita_income  median_rent  median_age  \n",
       "0                 4              23680          501        38.1  \n",
       "1                 6              32651          978        33.6  \n",
       "2                30              25358          747        36.3  \n",
       "3                 7              22170          480        37.5  \n",
       "4                38              29527         1119        35.4  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "demography_pd.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>region</th>\n",
       "      <th>value</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>alabama</td>\n",
       "      <td>4777326</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>alaska</td>\n",
       "      <td>711139</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>arizona</td>\n",
       "      <td>6410979</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>arkansas</td>\n",
       "      <td>2916372</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>california</td>\n",
       "      <td>37325068</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       region     value\n",
       "0     alabama   4777326\n",
       "1      alaska    711139\n",
       "2     arizona   6410979\n",
       "3    arkansas   2916372\n",
       "4  california  37325068"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "population_pd.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "State             object\n",
       "HighQ            float64\n",
       "HighQN             int64\n",
       "MedQ             float64\n",
       "MedQN              int64\n",
       "LowQ             float64\n",
       "LowQN              int64\n",
       "date      datetime64[ns]\n",
       "dtype: object"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prices_pd.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sort the data on state and date, then fill NA values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "prices_pd.sort(columns=['State', 'date'], inplace=True)\n",
    "prices_pd.fillna(method='ffill', inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finding mean, median, mode, variance, standard deviation for California"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mean\n",
    "\n",
    "arithmetic average of a range of values or quantities, computed by dividing the total of all values by the number of values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State</th>\n",
       "      <th>HighQ</th>\n",
       "      <th>HighQN</th>\n",
       "      <th>MedQ</th>\n",
       "      <th>MedQN</th>\n",
       "      <th>LowQ</th>\n",
       "      <th>LowQN</th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20098</th>\n",
       "      <td>California</td>\n",
       "      <td>248.77</td>\n",
       "      <td>12021</td>\n",
       "      <td>193.44</td>\n",
       "      <td>12724</td>\n",
       "      <td>193.88</td>\n",
       "      <td>770</td>\n",
       "      <td>2013-12-27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20863</th>\n",
       "      <td>California</td>\n",
       "      <td>248.74</td>\n",
       "      <td>12025</td>\n",
       "      <td>193.44</td>\n",
       "      <td>12728</td>\n",
       "      <td>193.88</td>\n",
       "      <td>770</td>\n",
       "      <td>2013-12-28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21577</th>\n",
       "      <td>California</td>\n",
       "      <td>248.76</td>\n",
       "      <td>12047</td>\n",
       "      <td>193.55</td>\n",
       "      <td>12760</td>\n",
       "      <td>193.60</td>\n",
       "      <td>772</td>\n",
       "      <td>2013-12-29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22291</th>\n",
       "      <td>California</td>\n",
       "      <td>248.82</td>\n",
       "      <td>12065</td>\n",
       "      <td>193.54</td>\n",
       "      <td>12779</td>\n",
       "      <td>193.80</td>\n",
       "      <td>773</td>\n",
       "      <td>2013-12-30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22801</th>\n",
       "      <td>California</td>\n",
       "      <td>248.76</td>\n",
       "      <td>12082</td>\n",
       "      <td>193.54</td>\n",
       "      <td>12792</td>\n",
       "      <td>193.80</td>\n",
       "      <td>773</td>\n",
       "      <td>2013-12-31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            State   HighQ  HighQN    MedQ  MedQN    LowQ  LowQN       date\n",
       "20098  California  248.77   12021  193.44  12724  193.88    770 2013-12-27\n",
       "20863  California  248.74   12025  193.44  12728  193.88    770 2013-12-28\n",
       "21577  California  248.76   12047  193.55  12760  193.60    772 2013-12-29\n",
       "22291  California  248.82   12065  193.54  12779  193.80    773 2013-12-30\n",
       "22801  California  248.76   12082  193.54  12792  193.80    773 2013-12-31"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "california_pd = prices_pd[prices_pd.State == \"California\"].copy(True)\n",
    "california_pd.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ca_sum = california_pd['HighQ'].sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ca_count = california_pd['HighQ'].count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mean weed price in CA is: 245.376124722\n"
     ]
    }
   ],
   "source": [
    "ca_mean = ca_sum / ca_count\n",
    "print \"Mean weed price in CA is:\", ca_mean"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Exercise: Find CA mean for 2013, 2014 & 2015 separately\n",
    "\n",
    "*Hint:* `california_pd.iloc[0]['date'].year`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Median\n",
    "\n",
    "Denotes value or quantity lying at the midpoint of a frequency distribution of observed values or quantities, such that there is an equal probability of falling above or below it. Simply put, it is the *middle* value in the list of numbers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "449"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ca_count"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If count is odd, the median is the value at (n+1)/2,\n",
    "\n",
    "else it is the average of n/2 and (n+1)/2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ca_highq_pd = california_pd.sort(columns=['HighQ'])\n",
    "ca_highq_pd.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Median price of weed in CA is: 245.31\n"
     ]
    }
   ],
   "source": [
    "ca_median = ca_highq_pd.HighQ.iloc[(ca_count) / 2]\n",
    "print \"Median price of weed in CA is:\", ca_median"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mode\n",
    "\n",
    "It is the number which appears most often in a set of numbers. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The most common price is CA, as indicated by its mode, is: 245.05\n"
     ]
    }
   ],
   "source": [
    "ca_mode = ca_highq_pd.HighQ.value_counts().index[0]\n",
    "print \"The most common price is CA, as indicated by its mode, is:\", ca_mode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Variance\n",
    "\n",
    "> Once two statistician of height 4 feet and 5 feet have to cross a river of AVERAGE depth 3 feet. Meanwhile, a third person comes and said, \"what are you waiting for? You can easily cross the river\"\n",
    "\n",
    "It's the average distance of the data values from the *mean*\n",
    "\n",
    "<img style=\"float: left;\" src=\"img/variance.png\" height=\"320\" width=\"320\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "california_pd['HighQ_dev'] = (california_pd['HighQ'] - ca_mean) ** 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Variance of High Quality weed prices in CA is: 2.98268628798\n"
     ]
    }
   ],
   "source": [
    "ca_HighQ_variance = california_pd.HighQ_dev.sum() / (ca_count - 1)\n",
    "print \"Variance of High Quality weed prices in CA is:\", ca_HighQ_variance"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Standard Deviation\n",
    "\n",
    "It is the square root of variance. This will have the same units as the data and mean. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Standard Deviation of High Quality weed prices in CA is: 1.72704553732\n"
     ]
    }
   ],
   "source": [
    "ca_HighQ_SD = np.sqrt(ca_HighQ_variance)\n",
    "print \"Standard Deviation of High Quality weed prices in CA is:\", ca_HighQ_SD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using Pandas built-in function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>HighQ</th>\n",
       "      <th>HighQN</th>\n",
       "      <th>MedQ</th>\n",
       "      <th>MedQN</th>\n",
       "      <th>LowQ</th>\n",
       "      <th>LowQN</th>\n",
       "      <th>HighQ_dev</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>449.000000</td>\n",
       "      <td>449.000000</td>\n",
       "      <td>449.000000</td>\n",
       "      <td>449.000000</td>\n",
       "      <td>449.000000</td>\n",
       "      <td>449.000000</td>\n",
       "      <td>449.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>245.376125</td>\n",
       "      <td>14947.073497</td>\n",
       "      <td>191.268909</td>\n",
       "      <td>16769.821826</td>\n",
       "      <td>189.783586</td>\n",
       "      <td>976.298441</td>\n",
       "      <td>2.976043</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>1.727046</td>\n",
       "      <td>1656.133565</td>\n",
       "      <td>1.524028</td>\n",
       "      <td>2433.943191</td>\n",
       "      <td>1.598252</td>\n",
       "      <td>120.246714</td>\n",
       "      <td>3.961134</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>241.840000</td>\n",
       "      <td>12021.000000</td>\n",
       "      <td>187.850000</td>\n",
       "      <td>12724.000000</td>\n",
       "      <td>187.830000</td>\n",
       "      <td>770.000000</td>\n",
       "      <td>0.000015</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>244.480000</td>\n",
       "      <td>13610.000000</td>\n",
       "      <td>190.260000</td>\n",
       "      <td>14826.000000</td>\n",
       "      <td>188.600000</td>\n",
       "      <td>878.000000</td>\n",
       "      <td>0.106357</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>245.310000</td>\n",
       "      <td>15037.000000</td>\n",
       "      <td>191.570000</td>\n",
       "      <td>16793.000000</td>\n",
       "      <td>188.600000</td>\n",
       "      <td>982.000000</td>\n",
       "      <td>0.729103</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>246.220000</td>\n",
       "      <td>16090.000000</td>\n",
       "      <td>192.550000</td>\n",
       "      <td>18435.000000</td>\n",
       "      <td>191.320000</td>\n",
       "      <td>1060.000000</td>\n",
       "      <td>4.435761</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>248.820000</td>\n",
       "      <td>18492.000000</td>\n",
       "      <td>193.630000</td>\n",
       "      <td>22027.000000</td>\n",
       "      <td>193.880000</td>\n",
       "      <td>1232.000000</td>\n",
       "      <td>12.504178</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            HighQ        HighQN        MedQ         MedQN        LowQ  \\\n",
       "count  449.000000    449.000000  449.000000    449.000000  449.000000   \n",
       "mean   245.376125  14947.073497  191.268909  16769.821826  189.783586   \n",
       "std      1.727046   1656.133565    1.524028   2433.943191    1.598252   \n",
       "min    241.840000  12021.000000  187.850000  12724.000000  187.830000   \n",
       "25%    244.480000  13610.000000  190.260000  14826.000000  188.600000   \n",
       "50%    245.310000  15037.000000  191.570000  16793.000000  188.600000   \n",
       "75%    246.220000  16090.000000  192.550000  18435.000000  191.320000   \n",
       "max    248.820000  18492.000000  193.630000  22027.000000  193.880000   \n",
       "\n",
       "             LowQN   HighQ_dev  \n",
       "count   449.000000  449.000000  \n",
       "mean    976.298441    2.976043  \n",
       "std     120.246714    3.961134  \n",
       "min     770.000000    0.000015  \n",
       "25%     878.000000    0.106357  \n",
       "50%     982.000000    0.729103  \n",
       "75%    1060.000000    4.435761  \n",
       "max    1232.000000   12.504178  "
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "california_pd.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    245.03\n",
       "1    245.05\n",
       "dtype: float64"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "california_pd.HighQ.mode()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Co-variance \n",
    "\n",
    "covariance as a measure of the (average) co-variation between two variables, say x and y. Covariance describes both how far the variables are spread out, and the nature of their relationship, Covariance is a measure of how much two variables change together. Compare this to Variance, which is just the range over which one measure (or variable) varies.\n",
    "\n",
    "<img style=\"float: left;\" src=\"img/covariance.png\" height=\"270\" width=\"270\">\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "#### Co-variance of weed price in California vs New York"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>State</th>\n",
       "      <th>HighQ</th>\n",
       "      <th>HighQN</th>\n",
       "      <th>MedQ</th>\n",
       "      <th>MedQN</th>\n",
       "      <th>LowQ</th>\n",
       "      <th>LowQN</th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20120</th>\n",
       "      <td>New York</td>\n",
       "      <td>351.98</td>\n",
       "      <td>5773</td>\n",
       "      <td>268.83</td>\n",
       "      <td>5786</td>\n",
       "      <td>190.31</td>\n",
       "      <td>479</td>\n",
       "      <td>2013-12-27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20885</th>\n",
       "      <td>New York</td>\n",
       "      <td>351.92</td>\n",
       "      <td>5775</td>\n",
       "      <td>268.83</td>\n",
       "      <td>5786</td>\n",
       "      <td>190.31</td>\n",
       "      <td>479</td>\n",
       "      <td>2013-12-28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21599</th>\n",
       "      <td>New York</td>\n",
       "      <td>351.99</td>\n",
       "      <td>5785</td>\n",
       "      <td>269.02</td>\n",
       "      <td>5806</td>\n",
       "      <td>190.75</td>\n",
       "      <td>480</td>\n",
       "      <td>2013-12-29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22313</th>\n",
       "      <td>New York</td>\n",
       "      <td>352.02</td>\n",
       "      <td>5791</td>\n",
       "      <td>268.98</td>\n",
       "      <td>5814</td>\n",
       "      <td>190.75</td>\n",
       "      <td>480</td>\n",
       "      <td>2013-12-30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22823</th>\n",
       "      <td>New York</td>\n",
       "      <td>351.97</td>\n",
       "      <td>5794</td>\n",
       "      <td>268.93</td>\n",
       "      <td>5818</td>\n",
       "      <td>190.75</td>\n",
       "      <td>480</td>\n",
       "      <td>2013-12-31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          State   HighQ  HighQN    MedQ  MedQN    LowQ  LowQN       date\n",
       "20120  New York  351.98    5773  268.83   5786  190.31    479 2013-12-27\n",
       "20885  New York  351.92    5775  268.83   5786  190.31    479 2013-12-28\n",
       "21599  New York  351.99    5785  269.02   5806  190.75    480 2013-12-29\n",
       "22313  New York  352.02    5791  268.98   5814  190.75    480 2013-12-30\n",
       "22823  New York  351.97    5794  268.93   5818  190.75    480 2013-12-31"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ny_pd = prices_pd[prices_pd['State'] == 'New York'].copy(True)\n",
    "ny_pd.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "ny_pd = ny_pd.ix[:,[1,7]]\n",
    "ny_pd.columns = ['NY_HighQ', 'date']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>NY_HighQ</th>\n",
       "      <th>date</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>20120</th>\n",
       "      <td>351.98</td>\n",
       "      <td>2013-12-27</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20885</th>\n",
       "      <td>351.92</td>\n",
       "      <td>2013-12-28</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>21599</th>\n",
       "      <td>351.99</td>\n",
       "      <td>2013-12-29</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22313</th>\n",
       "      <td>352.02</td>\n",
       "      <td>2013-12-30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22823</th>\n",
       "      <td>351.97</td>\n",
       "      <td>2013-12-31</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       NY_HighQ       date\n",
       "20120    351.98 2013-12-27\n",
       "20885    351.92 2013-12-28\n",
       "21599    351.99 2013-12-29\n",
       "22313    352.02 2013-12-30\n",
       "22823    351.97 2013-12-31"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ny_pd.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CA_HighQ</th>\n",
       "      <th>date</th>\n",
       "      <th>NY_HighQ</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>248.77</td>\n",
       "      <td>2013-12-27</td>\n",
       "      <td>351.98</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>248.74</td>\n",
       "      <td>2013-12-28</td>\n",
       "      <td>351.92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>248.76</td>\n",
       "      <td>2013-12-29</td>\n",
       "      <td>351.99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>248.82</td>\n",
       "      <td>2013-12-30</td>\n",
       "      <td>352.02</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>248.76</td>\n",
       "      <td>2013-12-31</td>\n",
       "      <td>351.97</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   CA_HighQ       date  NY_HighQ\n",
       "0    248.77 2013-12-27    351.98\n",
       "1    248.74 2013-12-28    351.92\n",
       "2    248.76 2013-12-29    351.99\n",
       "3    248.82 2013-12-30    352.02\n",
       "4    248.76 2013-12-31    351.97"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ca_ny_pd = pd.merge(california_pd.ix[:,[1,7]].copy(), ny_pd, on=\"date\")\n",
    "ca_ny_pd.rename(columns={\"HighQ\": \"CA_HighQ\"}, inplace=True)\n",
    "ca_ny_pd.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "346.9127616926502"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ny_mean = ca_ny_pd.NY_HighQ.mean()\n",
    "ny_mean"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CA_HighQ</th>\n",
       "      <th>date</th>\n",
       "      <th>NY_HighQ</th>\n",
       "      <th>ca_dev</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>248.77</td>\n",
       "      <td>2013-12-27</td>\n",
       "      <td>351.98</td>\n",
       "      <td>3.393875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>248.74</td>\n",
       "      <td>2013-12-28</td>\n",
       "      <td>351.92</td>\n",
       "      <td>3.363875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>248.76</td>\n",
       "      <td>2013-12-29</td>\n",
       "      <td>351.99</td>\n",
       "      <td>3.383875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>248.82</td>\n",
       "      <td>2013-12-30</td>\n",
       "      <td>352.02</td>\n",
       "      <td>3.443875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>248.76</td>\n",
       "      <td>2013-12-31</td>\n",
       "      <td>351.97</td>\n",
       "      <td>3.383875</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   CA_HighQ       date  NY_HighQ    ca_dev\n",
       "0    248.77 2013-12-27    351.98  3.393875\n",
       "1    248.74 2013-12-28    351.92  3.363875\n",
       "2    248.76 2013-12-29    351.99  3.383875\n",
       "3    248.82 2013-12-30    352.02  3.443875\n",
       "4    248.76 2013-12-31    351.97  3.383875"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ca_ny_pd['ca_dev'] = ca_ny_pd['CA_HighQ'] - ca_mean\n",
    "ca_ny_pd.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CA_HighQ</th>\n",
       "      <th>date</th>\n",
       "      <th>NY_HighQ</th>\n",
       "      <th>ca_dev</th>\n",
       "      <th>ny_dev</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>248.77</td>\n",
       "      <td>2013-12-27</td>\n",
       "      <td>351.98</td>\n",
       "      <td>3.393875</td>\n",
       "      <td>5.067238</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>248.74</td>\n",
       "      <td>2013-12-28</td>\n",
       "      <td>351.92</td>\n",
       "      <td>3.363875</td>\n",
       "      <td>5.007238</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>248.76</td>\n",
       "      <td>2013-12-29</td>\n",
       "      <td>351.99</td>\n",
       "      <td>3.383875</td>\n",
       "      <td>5.077238</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>248.82</td>\n",
       "      <td>2013-12-30</td>\n",
       "      <td>352.02</td>\n",
       "      <td>3.443875</td>\n",
       "      <td>5.107238</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>248.76</td>\n",
       "      <td>2013-12-31</td>\n",
       "      <td>351.97</td>\n",
       "      <td>3.383875</td>\n",
       "      <td>5.057238</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   CA_HighQ       date  NY_HighQ    ca_dev    ny_dev\n",
       "0    248.77 2013-12-27    351.98  3.393875  5.067238\n",
       "1    248.74 2013-12-28    351.92  3.363875  5.007238\n",
       "2    248.76 2013-12-29    351.99  3.383875  5.077238\n",
       "3    248.82 2013-12-30    352.02  3.443875  5.107238\n",
       "4    248.76 2013-12-31    351.97  3.383875  5.057238"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ca_ny_pd['ny_dev'] = ca_ny_pd['NY_HighQ'] - ny_mean\n",
    "ca_ny_pd.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Covariance of the High Quality weed prices in CA and NY is: 5.91681496729\n"
     ]
    }
   ],
   "source": [
    "ca_ny_cov = (ca_ny_pd['ca_dev'] * ca_ny_pd['ny_dev']).sum() / (ca_count - 1)\n",
    "print \"Covariance of the High Quality weed prices in CA and NY is:\", ca_ny_cov"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Using Pandas built-in function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CA_HighQ</th>\n",
       "      <th>NY_HighQ</th>\n",
       "      <th>ca_dev</th>\n",
       "      <th>ny_dev</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CA_HighQ</th>\n",
       "      <td>2.982686</td>\n",
       "      <td>5.916815</td>\n",
       "      <td>2.982686</td>\n",
       "      <td>5.916815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NY_HighQ</th>\n",
       "      <td>5.916815</td>\n",
       "      <td>12.245147</td>\n",
       "      <td>5.916815</td>\n",
       "      <td>12.245147</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ca_dev</th>\n",
       "      <td>2.982686</td>\n",
       "      <td>5.916815</td>\n",
       "      <td>2.982686</td>\n",
       "      <td>5.916815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ny_dev</th>\n",
       "      <td>5.916815</td>\n",
       "      <td>12.245147</td>\n",
       "      <td>5.916815</td>\n",
       "      <td>12.245147</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          CA_HighQ   NY_HighQ    ca_dev     ny_dev\n",
       "CA_HighQ  2.982686   5.916815  2.982686   5.916815\n",
       "NY_HighQ  5.916815  12.245147  5.916815  12.245147\n",
       "ca_dev    2.982686   5.916815  2.982686   5.916815\n",
       "ny_dev    5.916815  12.245147  5.916815  12.245147"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ca_ny_pd.cov()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Correlation\n",
    "\n",
    "Extent to which two or more variables fluctuate together. A positive correlation indicates the extent to which those variables increase or decrease in parallel; a negative correlation indicates the extent to which one variable increases as the other decreases.\n",
    "\n",
    "<img style=\"float: left;\" src=\"img/correlation.gif\" height=\"270\" width=\"270\">\n",
    "\n",
    "<br>\n",
    "<br>\n",
    "<br>\n",
    "\n",
    "#### Finding correlation between weed prices in New York and California"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Correlation between weed prices in NY and CA: 0.979043961106\n"
     ]
    }
   ],
   "source": [
    "ca_highq_std = ca_ny_pd.CA_HighQ.std()\n",
    "ny_highq_std = ca_ny_pd.NY_HighQ.std()\n",
    "\n",
    "ca_ny_corr = ca_ny_cov / (ca_highq_std * ny_highq_std)\n",
    "print \"Correlation between weed prices in NY and CA:\", ca_ny_corr"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>CA_HighQ</th>\n",
       "      <th>NY_HighQ</th>\n",
       "      <th>ca_dev</th>\n",
       "      <th>ny_dev</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CA_HighQ</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.979044</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.979044</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>NY_HighQ</th>\n",
       "      <td>0.979044</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.979044</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ca_dev</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.979044</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.979044</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ny_dev</th>\n",
       "      <td>0.979044</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.979044</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          CA_HighQ  NY_HighQ    ca_dev    ny_dev\n",
       "CA_HighQ  1.000000  0.979044  1.000000  0.979044\n",
       "NY_HighQ  0.979044  1.000000  0.979044  1.000000\n",
       "ca_dev    1.000000  0.979044  1.000000  0.979044\n",
       "ny_dev    0.979044  1.000000  0.979044  1.000000"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ca_ny_pd.corr()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Correlation != Causation\n",
    "\n",
    "correlation between two variables does not necessarily imply that one causes the other.\n",
    "\n",
    "\n",
    "<img style=\"float: left;\" src=\"img/correlation_not_causation.gif\" height=\"570\" width=\"570\">"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
